Apparatus and method for editing

ABSTRACT

An editing method comprises: generating, from a material data which is generated according to a first time and in which a first time range is specified, a playback data that is to be played back according to a second time different from the first time; and locating a second time range, which includes the material data within the first time range, in the playback data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The application is a United States National Stage Application under 35U.S.C. §371 of International Patent Application No. PCT/JP2009/003066,filed Jul. 2, 2009, which claims the benefit to Japanese Application No.2009-068566, filed Mar. 19, 2009, each of which is incorporated byreference into this application as if fully set forth herein.

TECHNICAL FIELD

The present invention relates to an editing apparatus, editing method,editing program and data structure.

BACKGROUND ART

There conventionally exist, for video editions, an approach in whicheditions are performed on a videotape or film on which video or imageshave been recorded and an approach in which video or image data recordedon a videotape or film is converted to digital data for editions.

In the case of performing editions on a videotape or film on which videoor images have been recorded, it is necessary to perform editionssequentially for a time axis of video or image data that is material(which time axis will be referred to as “material time” hereinafter),while reading temporal information that is called time code. Therefore,the editing operation is linear for the material time. In contrast,video or image data as converted to digital data, if stored in a harddisc or the like of a personal computer, workstation or the like, can beflexibly edited in addition to the playback direction and playback timeby accessing the stored video data on a random basis without taking thematerial time into account. Therefore, the editing operation may benonlinear for the material time. For this reason, an apparatus fordigitally editing video data is also called “nonlinear editingapparatus”. If such a nonlinear editing apparatus is used, then a widevariety of editions can be performed, beyond comparison with theconventional tape editions using videotapes, without having to take thetime code into account; for example, if some midway scene becomesunnecessary, the following scenes can be shifted forward.

In the case of using such a nonlinear editing apparatus, encoded videostored, for example, in a hard disc can be read therefrom, decoded,reproduced, and then edited while being viewed by use of a monitor.There conventionally exists a nonlinear video editing apparatus that canreproduce video data, which is to be edited, to allow the startingposition (in-point) and ending position (out-point) of a scene, which isto be cut out, to be set at the corresponding positions of the videodata, while allowing the video of the video data to be viewed. Thesein-point and out-point are associated with the video data and held asediting information, and a clip of the scene as cut out is formed basedon this editing information. It should be noted that in the nonlinearvideo edition, there is no need to actually copy or cut any originalvideo data in order to form clips and during a clip playback, a clip isspecified, thereby referring to the editing information and decoding andreproducing the video data ranging from the in-point to the out-point.

Some types of nonlinear editing apparatuses can use an editing function,which is called “clip marker”, to place marks at any desired scenepositions of the video and audio of a clip. Further, patent literature 1discloses an editing system wherein an editing function, which is called“time remapping”, is used to flexibly change the playback rate of thevideo and audio of a clip to any desired rate and display the status ofthe change in the playback rate on a user interface. The playback ratecan be changed by use of, for example, a key frame specified by a useror editor and a set of mathematical expressions of line segmentinterpolations and Bezier interpolations utilizing the key frame.

According to the conversional method, if video data that is material(which will be referred to as “material data” hereinafter) is subjectedto editions including a time remapping conversion process to generatevideo data to be played back (which will be referred to as “playbackdata” hereinafter), then it is relatively easy to cause marks placed inthe material data and indicating given “positions” on the material timeto be reflected in the playback data. However, in a case where marksdefining a given “range” on the material time are placed in the materialdata, if a time remapping conversion from the material time to theplayback time is simply applied, the change in the time interval betweenthe starting and ending points of the range is not taken into account,with the result that the range indicated by the marks placed in thematerial data may not correctly be reflected in the playback data.

FIGS. 1A-1C are graphs showing correspondences of clip markers betweenthe material time and the playback time. In FIGS. 1A-1C, marks placed atgiven positions of the material data are shown as clip markers a, b, c,d and e. FIG. 1A illustrates a case where a playback time is set suchthat a playback is performed at the same playback rate as the materialtime. In the example of FIG. 1A, as shown by solid lines, there exitonly single playback times a′ and b′ that correspond to the clip markersa and b of the material time, respectively, while as shown by dottedlines, there also exit only the single material times a and b thatcorrespond to the playback times a′ and b′, respectively. FIG. 1Billustrates a case where a playback time is set such that a playback isperformed with the playback rate of the material time varied. In theexample of FIG. 1B, as shown by solid lines, for example, there exist aplurality of playback times d1′, d2′ and d3′ corresponding to thematerial time denoted by the clip marker d. FIG. 1C illustrates anothercase where a playback time is set such that a playback is performed withthe playback rate of the material time varied. In the example of FIG.1C, there exists no playback time that corresponds to the material timedenoted by the clip marker e. However, even in such a case, therenecessarily exits a single material time corresponding to a time of theplayback time denoted by, for example, f.

In the case where the material time is the same as the playback time(FIG. 1A), the positions of the playback time correspond to therespective positions of the material time in a one-to-one relationship.In contrast, in the cases where the playback time is set such that theplayback is performed with the playback rate of the material time variedas shown in FIGS. 1B and 1C, there may exist a plurality of playbacktimes corresponding to the position of a material time, or no playbacktime may exit that corresponds to the position of a material time.

As previously stated, in a case where clip markers are used to define agiven “range” of the material data and then a time remapping conversionfrom the material time to the playback time is simply applied, thechange in the time interval between the starting and ending points ofthe range is not taken into account, with the result that the rangedefined by the clip markers placed in the material data may notcorrectly be reflected in the playback data. For example, in a casewhere a range is specified in such a manner that the clip marker d shownin FIG. 1B serves as the starting or ending point of the specifiedrange, the corresponding playback times d1′, d2′ and d3′ can be knownindeed but the time change between d1′ and d2′ and the time changebetween d2′ and d3′ are not taken into account. For this reason, itcannot be determined whether or not the ranges between d1′ and d2′ andbetween d2′ and d3′ in the playback time correspond to the rangespecified in the material time.

Therefore, in a case where the material data is a recorded data of, forexample, the TV broadcast of a succor game and a range of a goal scenein the game is specified by use of clip markers, even if the goal sceneas specified by use of the clip markers is to be played back, by use ofthe playback data, after a time remapping conversion process, then itcannot correctly be determined to which range of the playback data assubjected to the time remapping conversion process the range of the goalscene as specified in the material data corresponds, with the resultthat only a portion of the goal scene may be played back or the goalscene may not be appropriately played back.

Thus, according to the conventional art, it was possible to determinethe mutual correspondence between any given “position” in the materialdata and a “position” at which the given position is reflected in theplayback data as subjected to a time remapping conversion process.However, it was difficult to determine the mutual correspondence betweenany given “range” specified in the material data and a “range” in whichthe given range specified in the material data is reflected in theplayback data as subjected to a time remapping conversion process.

CITATION LIST Patent Literature

PTL 1: United States Patent Application Publication No. 2008/0253735A

SUMMARY OF INVENTION

The present invention provides an editing apparatus, editing method,editing program and data structure that make it possible to determinethe mutual correspondence between any given “range” specified in thematerial data and a “range” in which the given range specified in thematerial data is reflected in the playback data as subjected to a timeremapping conversion process.

According to an aspect of the invention, an editing method of theinvention comprises: generating, from a material data which is generatedaccording to a first time and in which a first time range is specified,a playback data that is to be played back according to a second timedifferent from the first time; and locating a second time range, whichincludes the material data within the first time range, in the playbackdata.

For example, in a case where the second time is set such that theplayback data includes a reverse playback range of the material data, ifthe playback data is generated from the material data, a plurality ofsecond time ranges corresponding to the first time range of the materialdata may appear in the playback data. Even in such a case, according tothe invention, the second time range, which includes the material datawithin the first time range, in the playback data can be located, sothat the correspondence between the first time range specified in thematerial data and the second time range, in which the specified range ofthe material data is reflected, in the playback data can be determined.The term “first time” used herein means a time axis used when thematerial data is recorded and corresponds to, for example, the “materialtime” as described in this specification. The term “second time” usedherein means a time axis used during the playback of the playback dataand corresponds to, for example, the “playback time” as described inthis specification. The same is true with the claims of the subjectapplication and the specification thereof unless any particularexplanations are made to the contrary. The terms “material data” and“playback data” include “moving image data”, “moving image signals” and“clips” as described in this specification. The same is true with theclaims of the subject application and the specification thereof unlessany particular explanations are made to the contrary. The material datawithin the first time range and the playback data within the second timerange correspond to, for example, “clip” as described in thisspecification.

The editing method according to the invention may further comprisedisplaying, on the user interface, the second time range in the secondtime or, alternatively, displaying, on the user interface, both thefirst time range in the first time and the second time range in thesecond time. As a result, the user can visually recognize a range ofplayback data in which the first range specified in the material data isreflected.

Moreover, the editing method according to the invention may include afeature that the first time range of the material data includesparticular identification information, which is displayed on the userinterface when the second time range on the user interface isdesignated. According to this feature, for example, if a user operates apointer on the user interface to designate the second time range, thenthe particular identification information is displayed on the userinterface. As a result, the user can visually recognize that the secondtime range of the playback data the user designates by use of thepointer is a range in which the first time range of the material data isreflected.

The editing method according to the invention may further comprise:associating the first time range in the first time with the second timerange in the second time; displaying, on the user interface, the firsttime range in the first time and the second time range in the secondtime; changing the manner of the displaying of the second time rangeassociated with the first time range, when the first time range on theuser interface is designated; and changing the manner of the displayingof the first time range associated with the second time range, when thesecond time range on the user interface is designated. According to thisfeature, for example, if a user operates a pointer on the user interfaceto designate the first time range of the material data, the user canvisually recognize the second time range of the playback data in whichthe first time range of the material data is reflected. Conversely, ifthe user operates the pointer on the user interface to designate thesecond time range of the playback data, the user can visually recognizethe first time range of the material data which is reflected in thesecond time range of the playback data. Changing the manner of thedisplaying of the first time range or second time range includes, forexample, charging the color of the displaying thereof, causing thedisplaying thereof to flash, increasing or decreasing the size of thedisplaying thereof, etc.

According to another aspect of the invention, an editing apparatus ofthe invention comprises a processing part that executes: generating,from a material data which is generated according to a first time and inwhich a first time range is specified, a playback data that is to beplayed back according to a second time different from the first time;and locating a second time range, which includes the material datawithin the first time range, in the playback data.

According to yet another aspect of the invention, a computer program ofthe invention allows a computer to execute: generating, from a materialdata which is generated according to a first time and in which a firsttime range is specified, a playback data that is to be played backaccording to a second time different from the first time; and locating asecond time range, which includes the material data within the firsttime range, in the playback data.

According to yet another aspect of the invention, a data structure ofthe invention comprises: a material data which has been generatedaccording to a first time and in which a first time range has beenspecified; and a playback data which has been generated from thematerial data and is to be played back according to a second timedifferent from the first time and in which a second time range, whichincludes the material data within the first time range, has beenlocated.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a diagram showing the correspondences of clip markers betweenthe material time and the playback time.

FIG. 1B is a diagram showing the correspondences of clip markers betweenthe material time and the playback time.

FIG. 1C is a diagram showing the correspondences of clip markers betweenthe material time and the playback time.

FIG. 2 is a block diagram of a nonlinear editing apparatus according toan embodiment of the invention.

FIG. 3 is a flowchart illustrating processings executed by the nonlinearediting apparatus of FIG. 2.

FIG. 4 is a flowchart illustrating, in detail, the processing in StepS13 of FIG. 3.

FIG. 5 is a diagram illustrating a principle on which the nonlinearediting apparatus of FIG. 2 locates, from material time ranges, thecorresponding playback time ranges.

FIG. 6 is a table showing criteria for determining, from differentialvalues and second-order differential values at intersections on a graph,whether reference positions represented by the respective intersectionsare the starting or ending positions of ranges.

FIG. 7 is a diagram illustrating a principle on which the nonlinearediting apparatus of FIG. 2 locates, from playback time ranges, thecorresponding material time ranges.

FIG. 8A is a diagram for illustrating a time remapping operation usingan editing apparatus having a clip marker function.

FIG. 8B is a diagram for illustrating the time remapping operation usingthe editing apparatus having the clip marker function.

FIG. 8C is a diagram for illustrating the time remapping operation usingthe editing apparatus having the clip marker function.

FIG. 9 is a graph illustrating a correspondence of a clip marker betweenthe material time and the playback time, which corresponds to a displayscreen shown in FIG. 8C.

DESCRIPTION OF EMBODIMENTS

Embodiments of the invention will be described in detail below withreference to the drawings.

A nonlinear editing apparatus, which is an editing apparatus accordingto a first embodiment of the invention, will now be described withreference to FIG. 2. As shown in FIG. 2, the nonlinear editing apparatus1 according to the present embodiment comprises a video input part 11; adecoder 12; an input interface (shown as “INPUT I/F” in this figure) 13;an input part 13 a that is a mouse, a keyboard, a control device and/orthe like; a CPU (Central Processing Unit) 14; a ROM (Read Only Memory)15; a RAM (Random Access Memory) 16; an output interface (shown as“OUTPUT I/F” in this figure) 17; a display 17 a; an encoder 18; a harddisc 19; and a video output part 20.

The video input part 11 receives, for example, video data from a cameraor the like. The decoder 12 decodes the video data from the video inputpart 11. The input interface 13 receives user instructions entered byoperating the input part 13 a that is the mouse, keyboard, controldevice and/or the like. The CPU 14 receives the user instructions viathe input interface 13 and controls all of the processings executed bythe nonlinear editing apparatus 1. The ROM 15 stores information, suchas various application programs to be executed by the CPU 14; screendata including an edition screen, various standby screens and the like;default data that are initial setting data of various functions. The RAM16 is used as a working memory of the CPU 14. The output interface 17outputs, under control of the CPU 14, video as processed and otherinformation to the display 17 a. The encoder 18 encodes the video dataas processed by the CPU 14, and the video output part 20 outputs thevideo data as encoded by the encoder 18 to another apparatus. The harddisc 19 can store video as processed by the CPU 14, video received fromthe video input part 11, and other information. In the presentembodiment of the invention, the CPU 14 functions as a processing part.

Processings to be executed by the nonlinear editing apparatus 1according to the present embodiment will now be described. Specifically,locating from a material time range a corresponding playback time rangeand locating from a playback time range a corresponding material timerange will be described in detail.

<1. Locating, from Material Time Ranges, the Corresponding Playback TimeRanges>

It will now be described, with reference to FIG. 2 and FIGS. 3-5, howthe nonlinear editing apparatus 1 according to the present embodimentlocates, from material time ranges, the corresponding playback timeranges. FIGS. 3 and 4 are flowcharts illustrating how the nonlinearediting apparatus 1 according to the present embodiment locates frommaterial time ranges the corresponding playback time ranges, while FIG.5 is a diagram illustrating a principle on which the nonlinear editingapparatus 1 locates from material time ranges the corresponding playbacktime ranges, and showing, as one example, a case where clip markers areused to define ranges C and D. The processings described below areperformed under control of the CPU 14. It is assumed that a materialdata has already been stored in the hard disc 19.

As shown in FIG. 3, to begin with, one or more ranges are set in thematerial data (Step S11). Specifically, the input part 13 a is operatedby a user, and in accordance with user's instructions received via theinput interface 13, the CPU 14 sets, for example, ranges C and D in thematerial data by use of clip markers. The CPU 14 associates the clipmarker information of the ranges C and D set by use of the clip markerswith the material data already stored in the hard disc 19 and stores theassociated clip marker information in the hard disc 19.

Next, the material data is edited to generate playback data (Step S12).Specifically, the input part 13 a is operated by the user, and inaccordance with user's instructions received via the input interface 13,the CPU 14 subjects the material data stored in the hard disc 19 to atime remapping edition for generating the playback data obtained byplaying back the material data according to a playback time (“secondtime”) the time axis of which is different from the time axis of thematerial data (“material time” or “first time”); adds the correspondingplayback rate information and the like to the material data; andgenerates the playback data. The CPU 14 need not actually copy or cutthe material data and may associate the editing information with thematerial data stored in the hard disc 19 and store the associatedediting information in the hard disc 19.

Next, portions of the playback data that correspond to the material timeranges set in the material data are located (Step S13). In the exampleof FIG. 4, based on the material data, clip marker information andediting information stored in the hard disc 19, the CPU 14 locates, inthe playback data, a range c₁-c₂ as a portion corresponding to the rangeC set in the material data by use of the clip markers and ranges d₁-d₃and d₄-d₅ as portions corresponding to the range D also set in thematerial data by use of the clip markers.

Locating the portions of the playback data that correspond to the rangesset in the material data by use of the clip markers (Step S13) will nowbe described in detail with reference to FIG. 4.

First, the CPU 14 determines, as reference positions, positions of theplayback time that correspond to the starting and ending positions ofthe material time ranges (Step S131).

The CPU 14 executes this processing by calculating a relationshipbetween the material time and the playback time according to a functiondepicted as a curve in FIG. 5. The material data is exactly notcontinuous and is constituted by a plurality of frames spaced by givenintervals. For those portions of the ranges C and D in which no framesexist, the CPU 14 uses line segment interpolation to perform thecalculation for a linearly changing portion, while using Bezierinterpolation to perform the calculation for a portion that changes at ahigh curvature. Thus, in Step S13, the CPU 14 determines all of thepoints, at which the starting position of the range C intersects thegraph of the interpolation values of the key frames, to obtain areference position C1, while determining all of the points, at which thestarting position of the range D intersects the graph of theinterpolation values of the key frames, to obtain reference positionsD1, D3 and D4. The CPU 14 then determines all of the points, at whichthe ending position of the range C intersects the graph of theinterpolation values of the key frames, to obtain reference positions C2and C3, while determining all of the points, at which the endingposition of the range D intersects the graph of the interpolation valuesof the key frames, to obtain reference positions D2 and D5.

Next, the CPU 14 determines the playback directions of the material timeat the respective reference positions (Step S132).

For executing this processing, the CPU 14 calculates differential valuesand second-order differential values for all of the obtained referencepositions C1, D1, D3, D4, C2, D2, C3 and D5, and determines, based onwhether the differential values and second-order differential values arepositive or not, the playback directions at the respective referencepositions.

Then, the CPU 14 determines, based on the thus determined playbackdirections, the starting and ending positions of the playback ranges inthe playback time (Step S133).

FIG. 6 shows a table for use in determining, based on the differentialvalues and second-order differential values at the intersections on thegraph, whether the reference positions represented by the respectiveintersections are starting or ending positions.

In accordance with this table, the “starting position” and “endingposition” of each “playback range” are determined for all of theintersections. If an intersection is the “starting position” of the“playback range”, then it is determined that the positions from the“starting position” to the next intersection are “insides of theplayback range”, while if an intersection is the “ending position” ofthe “playback range”, then it is determined that the positions from the“ending position” to the next intersection are “outsides of the playbackrange”. For the last intersection, there exists no next intersection, sothat the terminating position of the graph is used. This determinationis repeated until the last intersection, thereby obtaining sets of“insides of the playback range”. It should be noted that for the sets of“insides of the playback range” thus obtained, if any ranges areadjacent to each other, the corresponding sets are coupled to eachother. In this way, it is possible to locate the ranges of the playbackdata (playback range) in which the ranges specified in the material dataare reflected.

As a concrete example, locating, from the material time ranges of theclip markers C and D, the corresponding playback ranges will bedescribed below.

<1-1. Locating, from a Material Time Range of Clip Marker Range C, theCorresponding Playback Time Range>

First, a point, at which the starting position of the clip marker rangeC intersects the graph of interpolation values of the key frames, isdetermined and denoted as C1. Next, the differential value of theintersection C1, that is, a gradient of the graph at the intersection C1is determined. The differential value of the intersection C1 is positiveand hence greater than zero, which means that the playback direction atthe intersection C1 is the positive direction of the material time. Theintersection C1 corresponds to the “starting position” of the clipmarker range C and its differential value is positive, so that thecorresponding playback time is recognized as the starting position c1 ofthe “playback range”.

Next, points, at which the ending position of the clip marker range Cintersects the graph of interpolation values of the key frames, aredetermined and denoted as C2 and C3.

The differential value of the intersection C2 is determined first. Thedifferential value of the intersection C2 is positive and hence greaterthan zero, which means that the playback direction at the intersectionC2 is the positive direction of the material time. The intersection C2corresponds to the “ending position” of the clip marker range C and itsdifferential value is positive, so that the corresponding playback timeis recognized as the ending position c2 of the “playback range”.

Next, the differential value of the intersection C3 is determined. Thedifferential value of the intersection C3 is zero, which means theintersection C3 is a local maximum or a local minimum. In order to makeit possible to determine whether the intersection C3 is a local maximumor a local minimum, the second-order differential value of theintersection C3 is determined. The second-order differential value ofthe intersection C3 is positive and hence greater than zero, whichclarifies that the intersection C3 is a local minimum. The intersectionC3 corresponds to the ending position of the clip marker range C and itssecond-order differential value is positive and hence the intersectionC3 is a local minimum, so that the corresponding playback time isrecognized as the ending position c3 (not shown) of the “playbackrange”.

The positions c1, c2 and c3 are rearranged in ascending order of theplayback time. The positions c1 to c2 are inside the range, thepositions c2 to c3 are outside the range, and the position c3 to theposition corresponding to the graph terminal are outside the range. As aresult, a playback range c1-c2, the starting position of which is c1 andthe ending position of which is c2, is located as a range of playbackdata in which the range C specified in the material data is reflected.

<1-2. Locating, from a Material Time Range of Clip Marker Range D, theCorresponding Playback Time Ranges>

First, points, at which the starting position of the clip marker range Dintersects the graph of interpolation values of the key frames, aredetermined and denoted as D1, D3 and D4.

Next, the differential value of the intersection D1 is determined. Thedifferential value of the intersection D1 is positive and hence greaterthan zero, which means that the playback direction at the intersectionD1 is the positive direction of the material time. The intersection D1corresponds to the “starting position” of the clip marker range D andits differential value is positive, so that the corresponding playbacktime is recognized as the starting position d1 of the “playback range”.

Then, the differential value of the intersection D3 is determined. Thedifferential value of the intersection D3 is negative and hence smallerthan zero, which means that the playback at the intersection D3represents a playback in the reverse direction of the material time. Theintersection D3 corresponds to the “starting position” of the clipmarker range D and its differential value is negative, so that thecorresponding playback time is recognized as the ending position d3 ofthe “playback range”.

Then, the differential value of the intersection D4 is determined. Thedifferential value of the intersection D4 is positive and hence greaterthan zero, which means that the playback direction at the intersectionD4 is the positive direction of the material time. The intersection D4corresponds to the “starting position” of the clip marker range D andits differential value is positive, so that the corresponding playbacktime is recognized as the starting position d4 of the “playback range”.

Next, points, at which the ending position of the clip marker range Dintersects the graph of interpolation values of the key frames, aredetermined and denoted as D2 and D5.

The differential value of the intersection D2 is determined. Thedifferential value of the intersection D2 is zero, which means theintersection D2 is a local maximum or a local minimum. Then, in order todetermine whether the intersection D2 is a local maximum or a localminimum, the second-order differential value of the intersection D2 isdetermined. The second-order differential value of the intersection D2is negative and hence smaller than zero, which clarifies that theintersection D2 is a local maximum. The intersection D2 corresponds tothe “ending position” of the clip marker range D and its second-orderdifferential value is negative and hence the intersection D2 is a localmaximum, so that the corresponding playback time is recognized as thestarting position d2 (not shown) of the “playback range”.

Then, the differential value of the intersection D5 is determined. Thedifferential value of the intersection D5 is positive and hence greaterthan zero, which means that the playback direction at the intersectionD5 is the positive direction of the material time. The intersection D5corresponds to the “ending position” of the clip marker range D and itsdifferential value is positive, so that the corresponding playback timeis recognized as the ending position d5 of the “playback range”.

Then, the positions d1 to d5 thus obtained are rearranged in ascendingorder of the playback time. The positions d1 to d2 are inside the range,the positions d2 to d3 are inside the range, the positions d3 to d4 areoutside the range, the positions d4 to d5 are inside the range, and theposition d5 to the position corresponding to the graph terminal areoutside the range. The ranges d1-d2 and d2-d3 are adjacent to each otherand hence are coupled to each other. As a result, a playback time ranged1-d3, the starting position of which is d1 and the ending position ofwhich is d3, and a playback time range d4-d5, the starting position ofwhich is d4 and the ending position of which is d5, are located asranges of the playback data in which the range D specified in thematerial data is reflected.

<2. Locating, from Playback Time Ranges, the Corresponding Material TimeRanges>

Locating, from playback time ranges, the corresponding material timeranges by use of the nonlinear editing apparatus 1 according to thepresent embodiment will now be described with reference to FIGS. 2, 3and 7. FIG. 7 is a diagram illustrating a principle of locating, fromplayback time ranges, the corresponding material time ranges. It isassumed that a material time range has been specified by use of clipmarkers A and B and that video data that is material (which will bereferred to as “material data” hereinafter) has already been stored inthe hard disc 19. The processings described below are performed undercontrol of the CPU 14.

As a precondition for locating, from playback time ranges, thecorresponding material time ranges, it is necessary that locating, fromthe material time ranges, the corresponding playback time ranges asdescribed above have already been done. This location is just what wasdone in Steps S11-S13 described above with reference to FIG. 3. Thepresent example will now be explained as to a case where, as shown inFIG. 7, the input part 13 a is operated by the user and the CPU 14 sets,in response to user's instructions received via the input interface 13,for example, a clip marker in-point A and a clip marker out-point B inthe material data. The CPU 14 associates clip marker information of theclip marker in-point A and out-point B and the like with the materialdata already stored in the hard disc 19 and then stores the associatedclip marker information in the hard disc 19. In this way, the rangebetween the clip markers A and B in the material time is specified as amaterial time range (first time range).

Subsequently to Step S13, the material data ranges within the materialtime range set in Step S11 are located, from the portions located inStep S13, based on the time changes of the material data (Step S14). Therelationship between the material time (the time axis of the materialdata) and the playback time (the time axis of the playback data) isshown in FIG. 7. In the example of FIG. 7, the CPU 14 locates, based onthe material data, clip marker information and editing informationstored in the hard disc 19, ranges A₁-B₁ and B₂-B₄ in the playback dataas the portions corresponding to the range set in the material data byuse of the clip markers A and B, and further locates, as the materialdata ranges within the material time range, ranges defined by theminimum and maximum values of the material time in those located ranges.More specifically, the minimum value of the material time in a locatedrange, that is, the shortest elapsed time is recognized as the startingposition of a material data range within the material time range, whilethe maximum value of the material time, that is, the longest elapsedtime is recognized as the ending position of a material data rangewithin the material time range.

The CPU 14 can execute the processings described above by calculatingthe material time and playback time in accordance with the functiondepicted as a curve in FIG. 7. The material data is exactly notcontinuous and is constituted by a plurality of frames spaced bypredetermined intervals. It is assumed that frames specified by the clipmarker in-point A and out-point B are key frames A and B. As to thoseportions of the ranges A₁-B₁ and B₂-B₄ in which no key frames exist, theCPU 14 uses, for example, line segment interpolation to calculateinterpolation values for a linearly changing portion, while using, forexample, Bezier interpolation to calculate interpolation values for aportion that changes at a high curvature. Then, the CPU 14 locates, asthe material data ranges, ranges between the minimum and maximum valuesof the material time taken by the interpolation values of key frames A₁,B₁, B₂ and B₄. This will be explained in more detail with reference tothe example shown in FIG. 7. A range A₁-B₁ (second time range) in theplayback time includes the whole material time range A-B in the materialtime; a range B₂-B₄ (second time range) in the playback time includes amaterial data range B₃-B that is a part of the material time range A-Bin the material time. According to the processing of the presentembodiment, it can be determined that the range A₁-B₁ in the playbacktime reflects the material data range A-B in the material time and thatthe range B₂-B₄ in the playback time reflects a material data range B₃-Bin the material time.

It should be noted that although the present embodiment is described ascalculating interpolation values by use of line segment interpolationand Bezier interpolation, the present invention is not limited to thisembodiment. It is apparent that any other interpolation method, if itcan be regarded as preferable in implementation, may be used instead.For example, spline interpolation may be used as such anotherinterpolation method.

An example of line segment interpolation and Bezier interpolation to beexecuted by the CPU 14 will now be described. First, the mathematicalexpression of line segment interpolation is as follows:MaterialTime=(key2_value−key1_value)*(pos−key1_pos)/(key2_pos−key1_pos)+key1_value  (1)

where “pos” is a playback time position; key1 is a neighbor key framepreceding the playback time position; key1_pos is the playback timeposition of key1; key1 value is the material time position of key1; key2is a neighbor key frame following the playback time position; key2_posis the playback time position of key2; and key2_value is the materialtime position of key2.

Next, the mathematical expressions of Bezier interpolation are asfollows:Playback time=t ³*key2_pos+3*t²*(1−t)*ctrl2_pos+3*t*(1−t)²*ctrl1_pos+(1−t)³*key1_pos;Material time=t ³*key2_value+3*t²*(1−t)*ctrl2_value+3*t*(1−t)²*ctrl1_value+(1−t)³*key1_value  (2)

where t is a parameter; ctrl1 is a control point 1 of Bezier; ctrl1_posis the playback time position of ctrl1; ctrl1_value is the material timeposition of ctrl1; ctrl2 is a control point 2 of Bezier; ctrl2_pos isthe playback time position of ctrl2; and ctrl2_value is the materialtime position of ctrl2.

Using the above line segment interpolation and Bezier interpolation, theCPU 14, in Step S13, locates, from the material data range defined bythe clip markers A and B, that is, from the material time range, boththe portion defined by the key frames A₁ to B₁ and the portion definedby the key frames B₂ to B₄ as playback data ranges including thematerial data within that material time range. Subsequently, in StepS14, the CPU 14 recognizes a position, at which the material timeexhibits the minimum value in each of the portions defined by the keyframes A₁ to B₁ and key frames B₂ to B₄ of the corresponding playbackdata, as the starting position of the respective material data range,and further recognizes a position, at which the material time exhibitsthe maximum value in each of the portions defined by the key frames A₁to B₁ and key frames B₂ to B₄ of the corresponding playback data, as theending position of the respective material data range. In this way, thematerial data ranges A-B and B₃-B are located which correspond to theplayback data ranges A₁-B₁ and B₂-B₄, respectively.

Display screens to be displayed on the display 17 a when a user operatesthe nonlinear editing apparatus 1 for editions will now be describedwith reference to FIGS. 8A-8C. The screens to be displayed on thedisplay 17 a transition, with the progress of the editing operation, asshown in FIGS. 8A-8C. The processings described below are executed,basically, by the CPU 14 that receives user's instructions and the likeentered via the input interface 13 and reads application programs, videodata and the like stored in the ROM 15.

FIG. 8A illustrates a situation in which a control screen 120 isoperated to display a menu window 127. As shown in FIG. 8A, a displayscreen 100 displayed on the display 17 a comprises an image screen 110and the control screen 120. In the control screen 120, a status screen123 is displayed which shows current settings. If the menu window 127 isoperated to click a clip mark setting (“Set clip mark In/Out”), thedisplay screen is changed to a display screen as shown in FIG. 8B, inwhich editing operations using clip markers and time remapping can bestarted.

In FIG. 8B, in addition to the status screen 123 that indicates thecurrent settings, a time line 124 that indicates the playback time ofthe material time, a time line cursor 124 a and a clip marker 125 aredisplayed in the control screen 120. FIG. 8B illustrates a situation inwhich the clip marker 125, with which the user sets an in-point (In) andan out-point (Out) to specify a material time range 125 a, is arrangedon the time line 124 and further the time line cursor 124 a is currentlyoperated by the user. In the example shown in FIG. 8B, it is arrangedthat the material time range 125 a specified as described above include,as its particular identification information, textual information of “OnClouds” and that if the time line cursor 124 a designates the materialtime range 125 a, “On Clouds” be displayed on the image screen 110. As aresult, the user can visually recognize that this image is an image ofthe material time range 125 a specified by the clip marker 125.

FIG. 8C shows a working screen, which is used for the user to performtime remapping, and illustrates a situation in which a time remappingsetting screen 126 appears in addition to the time line 124 and clipmarker 125. The time remapping setting screen 126 comprises a time linebar 126 a that indicates the playback time after the time remapping, anda setting bar 126 b that indicates settings made to the time line bar126 a.

FIG. 9 is a graph that illustrates a correspondence of the clip markerbetween the material time and the playback time and that corresponds tothe display screen shown in FIG. 8C. It can be seen from FIG. 9 thatchanging the edition rate by use of the time remapping setting screen126 causes curves to occur in the graph due to the time remapping.

The material time range 125 a of the material data set in the clipmarker 125 is shown as two playback time ranges 125 b and 125 c on thetime line 124 displayed in the display screen after the time remappingedition shown in FIG. 8C. Also in this screen, it is arranged that ifthe time line cursor 124 a designates the playback time ranges 125 b and125 c, “On Clouds” be displayed in the image screen 110. As a result,the user can visually recognize that this image is an image within thematerial time range 125 a specified by use of the clip marker 125.

Thus, the nonlinear editing apparatus 1 according to the presentembodiment generates, from a material data which is generated accordingto a material time that is a first time and in which a material timerange that is a first time range is specified, a playback data, which isto be played back according to a playback time that is a second timedifferent from the material time, and locates a playback time range,which includes the material data within the material time range, in theplayback data. For example, in a case of setting the playback time suchthat the playback data includes a reverse playback range that is a rangein which the material data is played back in a direction opposite to theplayback direction of the material data, if the playback data isgenerated from the material data, a plurality of playback time rangescorresponding to a material time range of the material data may appearin the playback data. For example, as shown in FIG. 8C, the two playbacktime ranges 125 b and 125 c may appear which correspond to the materialtime range 125 a shown in FIG. 8B. Even in such a case, according to thenonlinear editing apparatus 1 of the present embodiment, the playbacktime ranges 125 b and 125 c, which include the material data within thematerial time range 125 a, in the playback data can be located, so thatthe correspondence between the material time range specified in thematerial data and each of the playback time ranges, in which thatmaterial time range is reflected, in the playback data can bedetermined.

The nonlinear editing apparatus 1 according to the present embodimentmay be adapted to display, on the user interface, the playback timerange (the playback time ranges 125 b and 125 c of FIG. 8C) in theplayback time or, alternatively, both the material time range in thematerial time (the material time range 125 a of FIG. 8B) and theplayback time range (the playback time ranges 125 b and 125 c of FIG.8C) in the playback time. As a result, the user can visually recognizethe playback data range in which the material data range specified inthe material data is reflected.

Moreover, the nonlinear editing apparatus 1 according to the presentembodiment may be adapted such that the material time range of thematerial data includes particular identification information, which isdisplayed on the user interface, like “On Clouds” is displayed on theimage screen 110 as shown in FIGS. 8B and 8C, when the playback timerange on the user interface is designated. According to thisarrangement, for example, if the user operates a pointer on the userinterface to designate the playback time range, the particularidentification information is displayed on the user interface. As aresult, the user can visually recognize that the playback time range ofthe playback data which the user designates by use of the pointer is arange in which the material time range of the material data isreflected.

The nonlinear editing apparatus 1 according to the present embodimentmay be adapted to associate the material time range 125 a in thematerial time with the playback time ranges 125 b and 125 c in theplayback time in response to an operation of the time remapping settingscreen 126 of FIG. 8C; display, on the user interface, the material timerange 125 a in the material time and the playback time ranges 125 b and125 c in the playback time; change the manner of the displaying of theplayback time ranges 125 b and 125 c, with which the material time range125 a has been associated, when the material time range 125 a on theuser interface is designated; and change the manner of the displaying ofthe material time range 125 a associated with the playback time ranges125 b and 125 c when the playback time ranges 125 b and 125 c on theuser interface are designated.

As a result, according to the nonlinear editing apparatus 1 of thepresent embodiment, for example, if the user operates the pointer on theuser interface to designate a material time range of material data, theuser can visually recognize a playback time range of playback data inwhich the material time range of the material data is reflected. If theuser operates the pointer on the user interface to designate a playbacktime range of the playback data, the user can visually recognize amaterial time range of the material data which is reflected in theplayback time range of the playback data. In a case of changing themanner of the displaying of the material time range or playback timerange, for example, the color of the displaying thereof may be changed,the displaying thereof may be caused to flash, or the size of thedisplaying thereof may be increased or decreased.

As previously described, there is no need to actually copy or cut videodata to be edited, that is, material data, but editing information maybe associated with the material data stored in the hard disc 19 and thenstored therein, so that a combination of the editing information and thematerial data stored in the hard disc 19 can be a playback data.

Therefore, the data structure of the data according to the presentembodiment, for example, if including the material data and editinginformation, comprises: the material data which has been generatedaccording to a first time, that is, a material time and in which a firsttime range, that is, a material time range has been specified; and theplayback data which has been generated from the material data and is tobe played back according to a second time, that is, a playback timedifferent from the material time and in which a second time range, thatis, a playback time range, which includes the material data within thematerial time range, has been located.

Such data structure may be adapted such that the material time range ofthe material data includes particular identification information usedfor identifying the material time range, for example, like “On Clouds”.As a result, a computer or the like may be used to execute displaying,on the user interface, such particular identification information, forexample, like “On Clouds” displayed on the image screen 110 as shown inFIGS. 8B and 8C, when, for example, the material time range or playbacktime range is designated on the user interface.

Moreover, such data structure may be adapted such that a computer canexecute: associating a material time range in the material time with aplayback time range in the playback time as done in the time remappingsetting screen 126 of FIG. 8C; changing the manner of the displaying ofthe playback time range with which the material time range is associatedwhen the material time range on the user interface is designated; andchanging the manner of the displaying of the material time rangeassociated with the playback time range when the playback time range onthe user interface is designated.

The above detailed descriptions were made as to preferable embodimentsof the invention. However, the invention is not limited to thoseparticular embodiments, and various modifications and variations may becontrived within the scope of the invention described in the claims. Inthe embodiments of the invention described above, the display screendisplayed on the display 17 a was used as the user interface, but theinvention is not limited to these embodiments and various otherembodiments may be used instead. Moreover, the editing apparatusaccording to the invention may be operated by use of an editing programthat allows a general-purpose computer including a CPU and a memory tofunction as the various means described above. Such an editing programmay be distributed via a communication line or may be written into arecording medium such as CD-ROM or the like for distribution.

The invention claimed is:
 1. An editing method, comprising: setting oneor more time ranges in material data to be edited, the material datahaving been generated according to a first time axis of material time;subjecting the material data to a time remapping for generating playbackdata that is to be played according to a second time axis of playbacktime different from the first time axis, the time remapping being suchthat a plurality of second time ranges corresponding to a first timerange of the one or more time ranges may appear in the playback data;associating editing information with the material data; storing theassociated editing information along with the material data; andgenerating playback data that is to be played back from the materialdata and the stored associated editing information, wherein thegenerating comprises locating a second time range, which includes thematerial data within the first time-range, in the playback data, whereingenerating the playback data that is to be played back includes:calculating a relationship between the material time and the playbacktime according to a function depicted as a curve by graphinginterpolation values of key frames; determining, as reference positions,positions of the playback time that correspond to the starting andending positions of the one or more time ranges in the material data, bydetermining all of the points at which the starting position and theending position of the set one or more ranges intersect the graph of theinterpolation values of the key frames; and determining a playbackdirection of the material time at the respective reference positions bydifferentiation of the calculated relationship.
 2. The editing methodaccording to claim 1, further comprising displaying, on a userinterface, said second time range in said second time axis.
 3. Theediting method according to claim 2, wherein: said first time range ofsaid material data includes particular identification information; andsaid method further comprises displaying, on said user interface, saidparticular identification in when said second time range on said userinterface is designated.
 4. The editing method according to claim 2,further comprising: associating said first time range in said first timeaxis with said second time range in said second time axis; displaying,on said user interface, said first time range in said first time axisand said second time range in said second time axis; changing the mannerof the displaying of said second time range associated with said firsttime range, when said first time range on said user interface isdesignated; and changing the manner of the displaying of said first timerange associated with said second time range, when said second timerange on said user interface is designated.
 5. An editing apparatus,comprising: a processing part configured to: set one or more time rangesin material data to be edited, the material data having been generatedaccording to a first time axis of material time; subject the materialdata to a time remapping for generating playback data that is to beplayed according to a second time axis of playback time different fromthe first time axis, the time remapping being such that a plurality ofsecond time ranges corresponding to a first time range of the one ormore time ranges may appear in the playback data; associate editinginformation with the material data store the associated editinginformation along with the material data; and generate playback datathat is to be played back from the material data and the storedassociated editing information, wherein the playback data is generatedby locating a second time range, which includes the material data withinthe first time-range, in the playback data, wherein generating theplayback data that is to be played back includes: calculating arelationship between the material time and the playback time accordingto a function depicted as a curve by graphing interpolation values ofkey frames: determining, as reference positions, positions of theplayback time that correspond to the starting and ending positions ofthe one more time ranges in the material data, by determining all of thepoints at which the starting position and the ending position of the setone or more ranges intersect the graph of the interpolation values ofthe key frames; and determining a playback direction of the materialtime at the respective reference positions by differentiation of thecalculated relationship.
 6. The editing apparatus according to claim 5,further comprising a user interface on which to display said second timerange in said second time axis or alternatively both said first timerange in said first time axis and said second time range in said secondtime axis.
 7. The editing apparatus according to claim 6, wherein: saidfirst time range of said material data includes particularidentification information: and said processing part causes saidparticular identification information to be displayed on said userinterface when said second time range on said user interface isdesignated.
 8. The editing apparatus according to claim 6, wherein saidprocessing part is further configured to: associate said first timerange in said first time axis with said second time range in said secondtime axis; cause said first time range in said first time axis and saidsecond time range in said second time axis to be displayed on said userinterface; change the manner of the displaying of said second time rangeassociated with said first time range, when said first time range onsaid user interface is designated; and change the manner of thedisplaying of said first time range associated with said second timerange, when said second time range on said user interface is designated.9. A non-transitory machine-readable medium comprising instructionsexecutable by a computer program, wherein said computer program causes acomputer to: set one or more time ranges in material data to be edited,the material data having been generated according to a first time axisof material time; subject the material data to a time remapping forgenerating playback data that is to be played according to a second timeaxis of playback time different from the first time axis, the timeremapping being such that a plurality of second time rangescorresponding to a first time range of the one or more time ranges mayappear in the playback data; associate editing information with thematerial data; store the associated editing information along with thematerial data; and generate playback data that is to be played back fromthe material data and the stored associated editing information, whereinthe playback data is generated by locating a second time range, whichincludes the material data within the first time-range, in the playbackdata, wherein generating the playback data that is to be played backincludes: calculating a relationship between the material time and theplayback time according to a function depicted as a curve by graphinginterpolation values of key frames; determining, as reference positions,positions of the playback time that correspond to the starting andending positions of the one or more time ranges in the material data, bydetermining all of the points at which the starting position and theending position of the set one or more ranges intersect the graph of theinterpolation values of the key frames; and determining a playbackdirection of the material time at the respective reference positions bydifferentiation of the calculated relationship.
 10. The non-transitorymachine-readable medium according to claim 9, wherein said computerprogram further causes said computer to display, on a user interface,said second time range in said second time axis or alternatively bothsaid first time range in said first time axis and said second time rangein said second time axis.
 11. The non-transitory machine-readable mediumaccording to claim 10, wherein: said first time range of said materialdata includes particular identification information; and said computerprogram further causes said computer to execute causing said particularidentification information to be displayed on said user interface whensaid second time range on said user interface is designated.
 12. Thenon-transitory machine-readable medium according to claim 10, whereinsaid computer program further causes said computer to: associate saidfirst time range in said first time axis with said second time range insaid second time axis; cause said first time range in said first timeaxis and said second time range in said second time axis to be displayedon said user interlace; change the manner of the displaying of saidsecond time range associated with said first time range, when said firsttime range on said user interlace is designated; and change the mannerof the displaying of said first time range associated with said secondtime range, when said second time range on said user interface isdesignated.
 13. The method of claim 1, wherein the time remapping iscarried out such that the playback data may include reverse playback ofmaterial data.
 14. The method of claim 1, wherein determining playbackdirections at the respective reference positions includes calculatingdifferential values and second order differential values for each of thereference points.
 15. An editing apparatus comprising a processorconfigured to: define first and second segments of a material clip;define a material time range for the first and second segments; define aplayback time range for the first segment that is different from thematerial time range for the first segment; in response to defining theplayback time range for the first segment, define a playback time rangefor the second segment such that the sum of the playback time ranges ofthe first and second segments is equal to the sum of the material timeranges of the first and second segments wherein defining the playbackrange includes: calculating a relationship between material time andplayback time according to a function depicted as a curve by graphinginterpolation values of key frames: determining, as reference positions,positions of the playback time that correspond to the starting andending positions of the one or more time ranges in the material data, bydetermining all of the points at which the starting position and theending position of the e set one or more ranges intersect the graph ofthe interpolation values of the key frames; and determining a playbackdirection of the material time at the respective reference positions bydifferentiation of the calculated relationship; associate editinginformation with the material data; and store the associated editinginformation along with the material data.
 16. The apparatus of claim 15,wherein the material clip is associated with a rate, and wherein theplayback time range for the first segment is defined by adjusting aplayback rate of the first segment to a rate different front the rateassociated with the material clip.
 17. The apparatus of claim 16,wherein the processor is further configured to: increase the playbackrate of the second segment when the playback rate of the first segmentis decreased; and decrease the playback rate segment then the playbackrate of the first segment is increased.
 18. The apparatus of claim 16,wherein the processor is further configured to define the playback rateof the first segment as negative.
 19. The editing method according toclaim 1, further comprising displaying, on a user interface, both saidfirst time range in said first time axis and said second time range insaid second time axis.
 20. The editing method according to claim 1,wherein the playback direction is determined by calculating differentialvalues and second order differential values for each of the obtainedreference points.
 21. The editing method according to claim 1, furthercomprising: determining a starting position and an ending position fromthe reference points based on the determined reference points and theirdetermined playback directions.
 22. The editing method according toclaim 21, further comprising: determining a set of a starting positionand an ending position that are within the playback range.